1. Field of the Invention
This invention generally relates to software testing, and more specifically, to methods and systems for selecting test cases for testing software applications. Even more specifically, the preferred embodiment of the invention relates to such methods and systems that provide dynamic functional testing coverage based on Failure Dependency Graphs (FDGs).
2. Background Art
When testing software systems, developing an effective and efficient test suite is a complex problem. A suite is effective if it thoroughly covers combinations of inputs, actions, and sequences of actions, enhancing the likeliness it will encounter defects in the software. A suite is efficient if it provides such coverage without unnecessary redundancy, which would unduly increase the testing effort.
Typically, when a tester tests a software system, the tester or test program takes an action against the system being tested and receives a result or response back from the tested system. The received result is compared against an expected result to determine if the system is working correctly. The tester performs a series of these action/response sequences. Sometimes, the order of this sequence depends upon the responses which were received earlier in the sequence. The number of ways that the sequences may be combined is very large.
Also, for most actions where the tester stimulates the system in a step in the sequence, the number of potential inputs to that particular action is very large. So, there are at least two places where one runs into a large number of combinations which leads to a combinatorial explosion problem.
Conventionally, focus for test generation has been on techniques such as finite-state based testing, which uses a finite state representation of the system being tested to generate sequences of actions that may be used to test the system. The actions are determined by seeking to cover states, transitions between states, or some combination thereof. These methods suffer from disadvantages of dealing with combinatorial explosion, and are complex to optimize from an internal state-based viewpoint.
Some companies have developed suites of tools that facilitate various types and stages of testing. One such suite is the International Business Machines (IBM®) Rational® Functional Tester (RFT). RFT is an object-oriented automated test tool that allows testing to be preformed on a variety of different application programs. It encompasses several modules that facilitate in the overall testing procedure. It enables testers to generate or use custom test scripts with choices in scripting languages and development environment. RFT contains object technologies that enable record-and-playback functionality for Java™, .Net™, and other applications including web-based applications. It also provides testers with automation capabilities to perform data-driven testing activities.
For example, when a tester writes or records a test script within RFT, the tool generates a test object map for the application under test. This object map is similar to a blueprint which contains object properties. The object map provides flexibility by allowing testers to modify the object map by reusing it, changing it, or by adding more objects as required. Once the object map is established, testers can insert verification points into the script which act to confirm the state of the object across a build process or test process. This is the baseline data file which contains the expected results when the program performs as it should. When a test is completed, a tester can utilize RFT's Verification Point Comparator to analyze differences or update the baseline if the behavior of the object changes.
In addition, Rational® Manual Tester [TM] enables manual test authoring and execution tools while also providing the monitoring for all aspects of manual and automated testing from iteration to iteration. Other tools include the IBM/Tivoli® Identity Manager (ITIM), which is a web-based application for testing applications providing security measures to access, such as log in screens. ITIM addresses a need to test the web interface to check for various conditions such as heavy stress, load situations etc., where manual testing of such user interfaces requires an excessive amount of human data entry and is often impossible to meet the proposed deadline due to time and resource constraints.
Clearly, automation plays a vital role in software testing. With shortened test cycles, reduced resources and increased workloads, testers rely heavily on automation to complete their tasks in timely fashion. With all the variety of testing tools and suites of products available, it is often difficult for testers to implement these tools in an efficient and effective manner.
In most Enterprise Business Applications, there will be a large number of test cases often numbering in tens or even hundreds of thousands. The challenge with these kinds of applications is in deciding which and how many test cases (suite) to run after the daily (periodic) builds. This is because, given the time and hardware constraints, it is impossible to run all the test cases periodically. This scenario requires a solution that provides an optimum test coverage while keeping in mind the resource constraints and hence improving the probability of finding the defects early. The challenge is to defining what an “optimum coverage” would be. Current solutions focus on defining quantity and feature based test coverage. These manual or brute force methods fail to achieve early detection of defects since they are not based on any understanding of the application or the relationships between testcases or features.